07_Naive_Bayes

Tags: #◇CSINTSY #◇Computational Thinking #◇Computer Science #◇Artificial Intelligence #◇Machine Learning #◇Intelligent Systems #◇Probability #◇Data Analysis

---

◇Overview

Naive Bayes is a probabilistic classifier based on Bayes' Theorem. It is called "Naive" because it makes a strong assumption that features are inde...

◇Details / Theory

7. Naive Bayes Reviewer

◇Overview

Naive Bayes is a probabilistic classifier based on Bayes' Theorem. It is called "Naive" because it makes a strong assumption that features are independent of each other given the class. Despite this simplification, it works surprisingly well, especially for text classification (e.g., Spam Filtering).

◇Bayes' Theorem

$P(C|X) = \frac{P(X|C) \times P(C)}{P(X)}$
$P(C|X)$: Posterior Probability. The probability of Class $C$ given features $X$. (What we want to find). $P(X|C)$: Likelihood. The probability of features $X$ appearing given Class $C$. $P(C)$: Prior Probability. The initial probability of Class $C$ (before seeing data). $P(X)$: Evidence. The probability of features $X$ appearing. (Usually ignored in comparison since it's constant for all classes).

◇The "Naive" Assumption

We assume features $x_1, x_2, \dots, x_n$ are independent.
$P(X|C) = P(x_1|C) \times P(x_2|C) \times \dots \times P(x_n|C)$

◇Classification Rule

To classify a new instance $X$, we calculate the score for each class and pick the highest one:
$Class = \text{argmax}_C P(C) \prod_{i=1}^{n} P(x_i|C)$

◇Example: Spam Filter

Classes: Spam, Ham (Not Spam). Message: "Free Money". Calculation:
$Score(Spam) = P(Spam) \times P("Free"|Spam) \times P("Money"|Spam)$
$Score(Ham) = P(Ham) \times P("Free"|Ham) \times P("Money"|Ham)$ Compare scores. If $Score(Spam) > Score(Ham)$, classify as Spam.

◇Laplace Smoothing (Add-One Smoothing)

Problem: If a word (feature) never appears in the training data for a specific class, its probability $P(word|Class)$ will be 0. Consequence: Since we multiply probabilities, a single 0 will make the entire score 0, regardless of other evidence. Solution: Add a small count (usually 1) to all frequency counts.
$P(x_i|C) = \frac{count(x_i, C) + 1}{count(C) + V}$
$V$: Vocabulary size (total number of unique words).
This ensures no probability is ever exactly zero.

◇Solved Example: Spam Classification

Dataset:
Spam (3 messages): "Buy cheap meds", "Cheap money", "Buy money" Ham (2 messages): "Cheap meds", "Study hard"

Vocabulary: {Buy, Cheap, Meds, Money, Study, Hard} (Size $V=6$)

Priors:
$P(Spam) = 3/5 = 0.6$ $P(Ham) = 2/5 = 0.4$

Likelihoods (with Laplace Smoothing +1):
Spam Word Count: 6 words total.
Count("Cheap" | Spam) = 2.
$P("Cheap" | Spam) = \frac{2 + 1}{6 + 6} = \frac{3}{12} = 0.25$
Count("Money" | Spam) = 2.
$P("Money" | Spam) = \frac{2 + 1}{6 + 6} = \frac{3}{12} = 0.25$
Ham Word Count: 4 words total.
Count("Cheap" | Ham) = 1.
$P("Cheap" | Ham) = \frac{1 + 1}{4 + 6} = \frac{2}{10} = 0.2$
Count("Money" | Ham) = 0.
$P("Money" | Ham) = \frac{0 + 1}{4 + 6} = \frac{1}{10} = 0.1$

Test Message: "Cheap Money"

Calculate Scores:

1. Spam Score:
   $P(Spam) \times P("Cheap"|Spam) \times P("Money"|Spam)$
   $0.6 \times 0.25 \times 0.25 = 0.0375$

2. Ham Score:
   $P(Ham) \times P("Cheap"|Ham) \times P("Money"|Ham)$
   $0.4 \times 0.2 \times 0.1 = 0.008$

Conclusion:
Since $0.0375 > 0.008$, the message is classified as Spam.

---

◇Common Pitfalls

Forgetting Laplace smoothing when a feature value has zero occurrences in a class — without it, the entire probability becomes 0. Violating the "naive" independence assumption without awareness — Naive Bayes assumes features are conditionally independent given the class. Comparing raw products instead of normalizing — final classification compares $P(C) \prod P(x_i|C)$ across all classes.

◇Summary

TL;DR — Naive Bayes classifies by computing $P(C|X) \propto P(C) \prod P(x_i|C)$ for each class and picking the maximum. It assumes feature independence and uses Laplace smoothing to handle zero-count features.

◇Links

Related: Source / Reference:

---

Made by 2ru17

Please contact me through discord or my the university email for any problems regarding the reviewer. Thanks!

◇References & Navigation

• Previous Topic: ◇Decision Tree
• Next Topic: ◇K Means Clustering
• Related Notes: ◇Naive Bayes